coding , AEP and Typicality 6 - 3 6 . 3 Using AEP for Data Compression

where l(x) is the length of codeword c(x) for a symbol x ∈ X , and p(x) is the probability of the symbol. Intuitively, a good code should preserve the information content of an outcome. Since information content depends on the probability of the outcome (it is higher if probability is lower, or equivalently if the outcome is very uncertain), a good codeword will use fewer bits to encode a certain or high probability outcome and more bits to encode a low probability outcome. Thus, we expect that the smallest expected code length should be related to the average uncertainty of the random variable i.e. the entropy. We will show that entropy is the fundamental limit of data compression; i.e., ∀C : L(C) ≥ H(X) (source coding theorem) Instead of encoding individual symbols, we can also encode blocks of symbols together. A length n block code encodes n length strings of symbols together and is denotes by C(x1, . . . , xn) =: C(x ). First, we will show that there exists a length n block code (an impractical one) with expected code length for a symbol that is arbitrarily close to entropy as n → ∞. This argument is due to Shannon as presented in his seminal paper (available at http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html). But first we need to introduce some new concepts.